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University  of  Illinois  under  Exploratory  Development  Task  Area  ZF55. 522.902 
(Methodology  for  Develop ing/ Evaluating  Navy  Training  Programs),  Work  Unit 
Number  01.60  (Advanced  Computer-Based  Research).  It  was  jointly  funded 
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ment and  administration,  including  decision  critiera  required  for  their 
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Dr.  Montague  also  served  as  the  technical  contract  monitor  for  this  effort. 
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SUMMARY 


Problem 


Students  often  fail  to  learn  from  books  and  other  instructional  sources 
because  they  do  not  study  them  carefully  enough.  Additionally,  managing 
the  large  groups  of  students  being  trained  in  university  and  military  set- 
tings entails  considerable  logistics  problems.  These  issues  are  related  to 
problems  of  applying  modern  instructional  and  computer  technology  to  the 
management  of  large-scale  instruction  while  minimizing  on-line  computer 
use  and  costs  and  material  preparation  costs.  The  research  reported 
here,  the  final  in  a series  of  studies,  addresses  these  problems. 

Purpose 

The  purpose  of  the  project  was  to  evaluate  (experimentally)  the  computer- 
based  Course  Management  System  (CMS),  which  was  designed  to  integrate  books, 
computer  testing,  and  personnel  for  training.  At  the  core  of  the  CMS  is  a 
computer-assisted  instruction  study  management  system  (CAISMS),  which  was 
developed  and  evaluated  in  earlier  research.  CAISMS  was  designed  to  question 
students  intermittently  about  what  they  are  reading  to  maintain  deep  cogni- 
tive processing. 

Approach 

The  system  was  implemented  on  the  PLATO  IV  instructional  system  at  the 
University  of  Illinois  during  two  semesters  of  an  introductory  level  economics 
course.  The  first  semester  served  primarily  as  a trial  implementation  in 
which  procedures  were  established,  computer  programs  were  debugged,  etc. 

In  the  second  semester,  a group  of  360  experimental  students  received  instruc- 
tion through  the  CMS.  These  students  acquired  concepts  and  information  from 
individual  reading  while  CAISMS  maintained  their  attention  to  the  material 
and  monitored  their  progress.  A control  group  of  70  students  was  taught 
by  a traditional  lecture-discussion  format,  but  used  the  same  text  and  took 
the  same  final  test  as  the  experimental  group.  The  instructor  time  that 
was  not  needed  for  the  experimental  students  was  invested  in  remediation 
and  special  interest  seminars.  The  two  groups  were  compared  on  final  exam 
performance  and  attitudes  toward  the  training. 

Findings 


Analysis  of  final  exam  data  showed  that,  when  type  of  test  questions 
was  examined,  the  experimental  group  performed  better  than  the  control  group 
on  text-related  items;  and  the  control  group,  on  lecture-related  items. 

The  groups  did  not  differ  on  items  selected  from  a standardized  test. 

Student  reactions  to  the  computer-assisted  instruction  and  the  seminars 
were  favorable.  However,  it  was  found  that  the  seminars  could  overburden 
some  of  the  students,  particularly  if  course-imposed  deadlines  were  not 
implemented  to  maintain  their  progress  through  the  textual  material. 


Conclusions 

It  was  concluded  that  the  CMS  had  facilitated  the  study  of  text,  and 
that  student  attitudes  regarding  the  system  were  favorable.  The  computer 
was  effective  for  managing  students,  keeping  records,  and  providing  direct 
communication  and  interaction  between  student  and  instructor. 

The  seminars  were  implemented  smoothly,  and  appeared  to  achieve  the 
purpose  of  making  economic  theory  more  understandable  via  practical  situa- 
tions. 

Recommendations 

It  is  recommended  that  CMS  be  implemented  In  a military  setting  to 
establish  usable  techniques  that  will  permit  Navy  operational  training 
personnel  to  modify  standard  courses,  to  ascertain  developmental  costs, 
and  to  determine  the  effectiveness  of  the  system  using  military  students. 
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INTRODUCTION 


Problem 

Navy  training  has  typically  been  conducted  using  traditional  classroom 
techniques.  Group  lectures  are  presented,  students  are  assigned  study  text- 
books, and  infrequent  examinations  are  given.  However,  textbooks  are  often 
ineffective  because  students  do  not  spend  enough  time  studying  them.  Further, 
the  lecture,  the  traditional  means  of  educating  large  nubmers  of  students 
in  university  and  military  settings,  is  typically  dull,  ineffective,  and 
inefficient  (Lloyd,  Garlington,  Lawry,  Burgess,  Evler,  & Knowlton,  1972). 

Also,  managing  large  numbers  of  students  in  such  settings  entails  consider- 
able logistics  problems. 

The  fact  that  a course  can  be  successful  without  lectures  has  been  clearly 
established.  In  a recent  comprehensive  review  of  instructional  techniques,  it 
was  shown  that  students  under  individualized  instruction  consistently  scored 
higher  than  those,  under  traditional  lecture  or  lecture-discussion  methods 
(Parsons,  undated).  More  specifically,  in  a study  of  Navy  training,  it  was 
found  that,  by  using  individualized  instruction  instead  of  standard  class- 
room instruction,  training  time  required  was  reduced  by  about  50  percent 
(Carson,  Graham,  Harding,  Johnson,  Mayo,  & Salop,  1975). 

A Course  Management  System  (CMS)  has  been  developed  that  provides  in- 
dividualized instruction  to  students  enrolled  in  large  classes.  Typically, 
the  use  of  individualized  instruction  requires  that  course  materials  be  com- 
pletely redeveloped  because  of  changed  requirements.  However,  in  the  design 
of  CMS,  an  attempt  was  made  to  minimize  the  material  development  effort  re- 
quired by  integrating  the  use  of  textbooks,  computers,  and  live  instructors. 

If  experimental  evaluation  of  this  system  proves  to  be  successful,  a means 
will  be  provided  for  converting  standard  group-lecture  type  courses  to 
an  individualized  format,  while  minimizing  the  developmental  efforts  re- 
quired. 

Background 

CMS  has  evolved  from  a series  of  research  and  development  efforts  begin- 
ning with  the  development  of  a computer-assisted  instruction  study  manage- 
ment system  (CAISMS),  which  was  implemented  and  evaluated  in  previous 
research  (Alessi,  Anderson,  Anderson,  Biddle,  Dalgaard,  Paden,  Smock,  Surber, 

& Wietecha,  1974;  Anderson,  Anderson,  Dalgaard,  Paden,  Biddle,  Surber, 

& Alessi,  1975a). 

The  rationale  behind  the  development  of  CAISMS  was  that  learning  from 
text  involves  a series  of  student-initiated  processing  activities  that  seldom 
are  fully  effective  because  of  conditions  prevailing  in  large  courses.  There 
is  a range  of  evidence  available  that  shows  that  procedures  that  induce  mean- 
ingful processing  of  text  facilitate  learning  (e.g.,  Barclay,  1973).  Further, 
the  results  of  75  years  of  research  indicate  that  asking  people  questions  about 
what  they  are  doing  is  an  effective  way  of  managing  processing  activities 
(Anderson  & Biddle,  1975).  This  was  the  technique  integrated  into  CAISMS. 
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The  system  was  implemented  on  the  PLATO  IV  instructional  system  at 
the  University  of  Illinois  with  an  introductory  level  economics  course. 

Terminals  were  installed  in  the  library  so  that  student  operators  could  be 
near  an  environment  in  which  they  could  study.  The  instructional  procedure 
involved  a student  signing  in  on  the  terminal,  receiving  an  assignment, 
studying  the  material  in  the  nearby  workspace,  signing  in  again,  and  re- 
ceiving a quiz  covering  the  reading  materials.  If  the  student's  quiz  score 
was  75  percent  or  more,  he  could  progress  to  the  next  assignment  and  go 
through  the  cycle  again.  If  his  score  was  less  than  75  percent,  he  was 
directed  by  CAISMS  to  review  the  same  study  assignment  and  retake  the  quiz. 

The  quiz  scores  did  not  affect  the  student's  final  grade,  but  he  had  to 
pass  a certain  number  of  them  before  he  could  take  the  final  exam.  This 
contingency  made  it  almost  impossible  for  students  to  go  into  the  exam 
unprepared. 

The  quiz  items  were  developed  to  measure  how  well  the  student  compre- 
hended the  important  concepts  in  the  text,  as  judged  by  the  economist 
subject  matter  experts  on  the  project  team.  They  either  paraphrased  the 
language  in  the  text,  or  required  the  student  to  apply  concepts  and  prin- 
ciples to  examples  other  than  those  appearing  in  the  text.  These  types  of 
items  increase  the  probability  that  the  student  will  engage  in  meaningful 
cognitive  processing  (Watts  & Anderson,  1971;  Felker,  1974). 

During  the  Fall  semester  of  1973,  the  feasibility  of  CAISMS  was  tested 
as  a supplement  to  the  standard  lecture  and  discussion  sections.  A group 
of  75  students  was  asked  to  interact  with  PLATO  IV  terminals  to  receive 
all  reading  assignments.  Performance  was  evaluated  by  the  scores  the 
students  obtained  on  the  quizzes  and  the  off-line  examinations.  At  the 
end  of  the  semester,  questionnaires  were  administered  to  determine  student 
attitudes  toward  use  of  CAISMS.  Results  indicated  that  the  system  was 
feasible  to  administer  and  potentially  effective  in  producing  achievement 
gains  over  traditional  types  of  instruction.  Further,  it  was  rated  favor- 
ably by  a majority  of  the  students  (Alessi  et  al.,  1974;  Anderson,  Anderson, 

Dalgaard,  Wietecha,  Biddle,  Paden,  Smock,  Alessi,  Surber,  & Klemt,  1974). 

CAISMS  was  experimentally  evaluated  during  the  Spring  semester  of  1974 
(Anderson,  Anderson,  Dalgaard  et  al.,  1975a  and  b) . Approximately  200 
students  were  randomly  assigned  either  to  classes  using  CAISMS  or  to  control 
classes  using  traditional  methods.  All  classes  had  a seminar  lecture 
discussion  format,  used  the  same  textbook,  and  received  an  identical  battery 
of  achievement  tests  and  questionnaires  during  the  semester.  Results  indi- 
cated that  members  of  the  CAISMS  classes  scored  higher  on  achievement  tests 
than  members  of  the  control  classes.  Further,  the  attitudes  of  CAISMS 
students  regarding  their  method  of  instruction  were  more  positive  than  those 
of  the  control  students.  Attrition  rates  were  approximately  equal  in  the 
two  groups. 

The  Course  Management  System  (CMS)  was  designed  in  the  Summer  of  1974 
and  implemented  for  feasibility  testing  during  the  Fall  semester  of  that 
year  (Anderson,  Anderson,  Alessi,  Dalgaard,  Paden,  Biddle,  Surber,  & 

Smock,  1975a  and  b) . In  CMS,  students  are  expected  to  acquire  basic  in- 
formation and  concepts  primarily  from  individual  reading.  Their  attention 
to  the  material  is  maintained  and  their  progress  monitored  by  the  previously 
developed  CAISMS.  Since  lectures  and  standard  quiz  sections  are  not  in- 
cluded in  CMS,  instructors  can  devote  their  time  to  providing  remedial 
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feedback  for  students  having  trouble  in  mastering  core  curriculum  and  to 
teaching  seminars.  The  role  of  the  computer  (PI.AIO  IV)  was  to  manage 
study  behavior,  to  administer  on-line  achievement  tests,  to  provide  a 
message  exchange  system  for  students  and  instructors,  and  to  schedule 
group  tutorial  and  seminar  sessions. 

The  concept  of  using  seminars  as  a supplement  to  computer-assisted 
instruction  was  developed  during  the  early  stages  of  CMS.  The  idea  was 
that  students  should  be  provided  with  an  in-depth  look  at  one  or  more 
areas  of  practical  economics.  In  other  words,  a minicourse  would  be 
developed  that  would  allow  students  to  explore  an  economic  problem  in 
some  detail.  During  this  exploration,  they  would  use  their  economic 
skills  developed  through  their  work  with  the  textual  aspects  of  the 
course  to  investigate  an  economic  problem  (also  see  Dalgaard , Paden,  & 
Anderson,  1976) . 

Five  instructors  were  selected  from  the  teaching  staff  to  select  the 
topics  of  the  seminars,  to  develop  course  materials  and  procedures,  and 
to  teach  the  seminars. 

During  the  Fall  semester  of  1974,  a group  of  360  students  used  the 
CMS  while  procedures  were  being  established  and  the  computer  programs  were 
being  debugged.  Ten  seminars,  covering  a variety  of  topics  ranging  from 
Environmental  Economics  to  Consumerism,  were  offered  several  times  during 
the  semester.  They  were  4 weeks  in  length,  with  two  1-hour  meetings  held 
each  week.  A small  class  size  was  maintained  (10  to  15  students),  and 
seminar  performance  counted  as  one-third  of  the  course  grade.  CMS  feasi- 
bility was  established,  and  achievement  tests  showed  that  students  using 
CMS  performed  as  well  as  previous  students  who  had  been  instructed  under 
traditional  classroom  methods  (Anderson,  Anderson,  Alessi  et  al.,  1975a). 

Table  1 presents  a chronological  summary  of  the  project  and  identifies 
the  relevant  documentation. 

Purpose 

The  present  effort  is  a continuation  of  that  reported  by  Anderson, 
Anderson,  Alessi  et  al.,  1975a.  Its  objective  was  to  evaluate  experi- 
mentally the  effectiveness  of  the  CMS  within  a university  setting  where 
large  numbers  of  students  are  available.  It  is  expected  that  the  use  of 
CMS  will  provide  substantial  reductions  in  efforts  required  to  develop 
individualized  courses  and  in  costs  of  operating  large  courses,  as  well 
as  increased  effectiveness  in  terms  of  both  learning  time  required  and  per- 
formance. 
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METHOD 


Subjects 

A total  of  780  undergraduate  students  enrolled  in  the  introductory 
level  economics  class  at  the  University  of  Illinois  participated  in  the 
project.  This  number  included  the  360  students  enrolled  in  the  Fall 
semester  of  1974  (previously  described  under  Background)  and  420  students 
enrolled  in  the  Spring  semester  of  1975.  Of  the  latter  group,  350  were 
assigned  to  the  experimental  group  and  70,  to  the  control  group.  However, 
due  to  normal  attrition,  a total  of  only  373 — 319  in  the  experimental 
group  and  64  in  the  control  group — took  the  final  performance  examination 
and  was  included  in  the  final  analyses. 

In  subsequent  sections,  the  Fall  semester  of  1974  will  be  referred  to 
as  the  first  semester;  and  the  Spring  semester  of  1975,  as  the  second 
semester.  For  both  semesters,  all  students  used  the  same  introductory 
economics  text — Samuelson,  Economics  (9th  edition),  1973. 

Apparatus 

The  economics  course  was  managed  for  all  students  in  the  first  semester 
and  the  experimental  students  in  the  second  semester  by  the  PLATO  IV  instruc- 
tional system.  This  system  consists  of  a Control  Data  Corporation  Cyber 
73  computer  and  a number  of  remote  terminals,  which  include  a keyset  and 
a display  panel  that  looks  like  a television  screen.  Students  can  relay 
messages  to  the  computer  via  the  keyset  and  receive  messages  from  the 
computer  via  the  display  panel.  In  most  instances,  the  response  is  im- 
mediate. Detailed  information  on  this  system  is  provided  by  Stifle  (1972a, 
1972b,  1973). 

Procedure 

All  students  in  the  first  semester  and  the  experimental  students  in  the 
second  semester  were  instructed  by  the  CMS  on  the  PLATO  system  (CAISMS) . 

These  students  were  given  assignments  in  the  text  by  the  computer.  When 
they  finished  these  assignments,  they  were  required  to  take  short  practice 
quizzes  from  PLATO  to  determine  whether  they  comprehended  the  material. 

A total  of  63  quizzes  was  given,  covering  specific  sections  in  27  chapters 
of  the  text.  Although  the  student’s  final  grade  was  not  influenced  by  the 
grades  he  received  on  the  quizzes,  he  was  required  to  complete  the  appropri- 
ate quizzes  in  sequence  before  getting  his  next  assignment.  Moreover,  if 
he  received  poor  grades  on  the  quizzes,  he  was  obligated  to  seek  help 
from  a teaching  assistant  or  a tutor  (undergraduate  economic  majors  assigned 
to  help  teaching  assistants). 

After  the  CAISMS  student  had  passed  a specific  number  of  short  quizzes, 
he  was  notified  by  the  computer  that  he  was  eligible  to  take  a 1-hour 
graded  examination.  Three  of  these  exams  were  administered  by  PLATO 
during  the  semester.  To  assist  the  student  in  preparing  for  these  exams, 
a large  number  of  review  items  had  been  prepared  that  were  similar  in  for- 
mat and  language  to  those  on  the  test.  The  student  had  free  access  to  these 


items  after  he  had  become  eligible  to  take  the  exam.  The  computer  would 
display  a random  sample  of  10  items  to  the  student,  provide  feedback 
(correct  or  incorrect)  immediately  after  the  student  had  answered  an  item, 
and  summarize  the  student's  performance  after  he  had  answered  all  10  items. 
Students  could  try  as  many  samples  of  items  as  they  wished  before  taking  the 
exam. 

When  a graded  exam  had  been  completed,  the  computer  immediately  provided 
the  grade  obtained  to  the  student,  as  well  as  his  cumulative  record  and 
standing  in  the  course.  He  could  then  go  on  to  the  next  assignment  and 
go  through  the  process  again.  By  using  CAISMS,  students  could  progress 
at  their  own  pace.  It  was  possible  for  them  to  complete  the  course  in  as 
little  as  12  weeks. 

At  some  time  during  the  semester,  CAISMS  students  were  required  to  select 
at  least  one  seminar  on  topics  of  current  interest.  For  those  seminars 
requiring  specific  analytical  skills,  satisfactory  progress  to  a particular 
point  in  the  text  was  a prerequisite.  The  computer  provided  scheduling 
information  for  these  seminars,  as  well  as  for  remedial  sessions. 

As  indicated  previously,  the  seminars  were  4 weeks  in  length,  with 
two  1-hour  meetings  per  week,  and  included  from  10  to  15  students.  Most 
seminars  in  both  semesters  were  taught  by  the  same  five  instructors  who  had 
developed  them;  the  remainder  were  taught  by  student  tutors  and  other 
faculty  members.  However,  in  the  first  semester,  seminar  performance 
counted  as  one-third  of  the  final  grade;  and  in  the  second  semester, 
as  one-fifth.  Also,  since  students  in  the  first  semester  appeared  to  be 
falling  behind  in  studying  their  text,  deadlines  in  completing  text  work 
were  imposed  for  the  second  semester  to  ensure  that  students  maintained 
their  progress  toward  course  completion.  These  deadlines  also  meant  that 
students  almost  always  entered  a seminar  just  after  completing  a section 
of  the  course  leading  to  a 1-hour  graded  exam.  As  a result,  some  seminars 
during  the  second  seminar  (10  of  22)  were  held  on  an  irregular  basis  rather 
than  meeting  continuously  for  4 weeks. 

The  70  control  students  in  the  second  semester  were  taught  by  the  same 
instructors  who  were  responsible  for  the  experimental  students,  but  without 
the  assistance  of  CMS.  They  interacted  with  the  instructor  in  a lecture- 
discussion  setting,  and  were  given  the  same  assignments  and  1-hour  graded 
exams  as  the  experimental  students.  However,  they  did  not  take  the  numerous 
short  quizzes  on  specific  parts  of  the  text  nor  were  they  able  to  partici- 
pate in  seminars.  Rather,  they  were  exposed  intensively  to  only  one  topic — 
the  Economics  of  Inflation. 

Measuring  Student  Performance 

As  indicated  previously,  the  performance  of  students  using  CMS  in  the 
first  semester  was  based  on  scores  obtained  in  the  course  examination. 

For  the  second  semester,  the  performance  of  both  the  experimental  and 
control  groups  was  measured  by  the  scores  obtained  on  a final  examination 
containing  70  items.  Of  this  number,  30  items  were  text  related  (TEXT), 

30  were  selected  from  standardized  items  from  a test  prepared  by  the  Joint 
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Council  on  Economic  Education  (JC),  and  10  were  interpretive  exercises 
(IE)  designed  by  the  instructors.  Also,  for  all  second  semester  students, 
data  were  obtained  regarding  their  high  school  percentile  rank,  credit 
hours  completed  in  the  undergraduate  program,  undergraduate  grade  point 
average,  and  student  class  standing  (e.g.,  freshman). 

Determining  Student  Attitudes 

Near  the  end  of  the  second  semester,  experimental  group  students  were 
administered  a questionnaire  via  PLATO.  Questions  were  asked  regarding 
their  attitudes  toward  the  course  and  computer-assisted  instruction.  For 
each  question  presented  on  the  screen,  students  selected  that  alternative 
which  most  closely  reflected  their  attitude  toward  the  particular  topic 
presented . 

As  part  of  the  procedures  to  determine  the  effectiveness  of  the  seminars, 
experimental  group  students  in  both  semesters  were  administered  paper-and- 
pencil  evaluation  questionnaires  at  the  end  of  each  seminar.  In  addition, 
during  the  second  seminar,  experimental  group  students  were  administered 
course  evaluation  questionnaires  via  PLATO.  These  questionnaires  included 
questions  designed  to  elicit  student  attitudes  toward  the  individual 
seminars  and  the  instructors  who  conducted  the  same  seminars  both  semesters, 
and  provided  an  opportunity  for  making  open-ended  comments.  Questions  con- 
cerning instructors  were  included  to  determine  any  interactions  between 
instructors  and  semesters. 
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RESULTS  AND  DISCUSSION 


The  performance  of  the  experimental  and  control  groups  on  the  final 
examination  and  their  responses  to  the  questionnaires  provide  the  primary 
bases  for  evaluating  the  effectiveness  and  adequacy  of  the  Course  Management 
System  (CMS). 

Student  Performance 

Table  2 provides  the  mean  scores  obtained  by  second  semester  experimental 
and  control  students  on  the  text  related  (TEXT),  standardized  Joint  Council 
on  Economic  Education  (JC) , and  interpretative  exercise  (IE)  items.  As 
shown,  the  experimental  group  scored  significantly  higher  on  the  TEXT  items; 
and  the  control  group,  on  the  IE  items.  There  was  no  significant  difference 
between  the  two  groups  on  the  standardized  JC  items. 


Table  2 

Performance  of  Second  Semester  Experimental 
and  Control  Students 


TEXT 

Items  (30) 

JC 

Items  (30) 

IE 

Items  (10) 

Total 

Group 

Mean 

S.D. 

Mean  S.D. 

Mean 

S.D. 

Mean  S.D. 

Experimental 
(N  = 319) 

20.45 

3.55 

20.52  4.11 

6.19 

1.83 

46.52  7.84 

Control 
(N  = 64) 

18.86 

4.15 

20.01  4.98 

6.97 

2.25 

45.43  9.79 

L 

Experimental 
vs.  Control 

t = 3.2, 

p < .002 

t = .87, 
p < .38 

t = -2.99, 
p < .003 

t * .84, 
p < .41 

Since  students  had  not  been  randomly  assigned  to  experimental  and  control 
groups,  regression  analyses  were  performed  to  correct  for  possible  between- 
group  differences.  Table  3 shows  the  means  and  correlation  coefficients 
between  the  three  achievement  variables  (i.e.,  scores  obtained  on  TEXT,  JC, 
and  IE  items)  and  the  five  predictor  variables  (i.e.,  high  school  percen- 
tile rank  (HSRANK) , cumulative  hours  in  undergraduate  degree  program  (HOURS), 
undergraduate  grade  point  average  (GPA) , student  year  in  school  classification 
(CLASSIF) , and  treatment  group  (experimental  or  control)  (TREAT)).  The 
most  striking  feature  of  the  data  pattern  shown  in  Table  3 is  the  consis- 
tently high  positive  correlation  between  measures  of  GPA  and  the  three 
achievement  variables. 
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Table  3 


Correlations  of  Predictor  and  Achievement  Variables 


Var 

iable 

Mean 

Correlation  Coefficients 

1 

2 

3 

4 

5 

6 

7 

8 

1. 

HSRANK 

84.2 

1.0 

2. 

HOURS 

47.3 

.04 

1.0 

3. 

GPA 

3.85 

.42 

.05 

1.0 

4. 

CLASSIF 

1.7 

-.05 

.70 

.15 

1.0 

5. 

TREAT 

1.8 

.00 

.30 

-.06 

-.36 

1.0 

6. 

TEXT 

20.2 

.32 

.01 

.55 

.11 

.16 

1.0 

7. 

JC 

20.4 

.28 

.05 

.56 

.13 

.05 

.67 

1.0 

8. 

IE 

6.3 

.26 

.15 

.40 

.21 

-.13 

.42 

.47 

1.0 

Note.  Number  of  students  = 367. 


Table  4 shows  the  standardized  multiple  regression  weights  obtained 
when  the  five  predictor  variables  were  used  simultaneously  to  predict  each 
of  the  achievement  variables.  The  multiple  regressions  for  predicting  TEXT, 
JC,  and  IC  are  .61,  .58,  and  .45  respectively.  The  predictor  variable  that 
contributes  most  to  the  prediction  scheme  for  each  of  the  three  achievement 
variables  is  GPA.  Thus,  knowledge  of  how  well  a student  performed  in 
other  courses  is  the  best  predictor  of  how  he  performed  in  the  introductory 
economics  class.  The  second  best  predictor  on  the  TEXT  variable  was  TREAT, 
or  knowledge  of  whether  the  student  belonged  to  the  experimental  (CMS) 
or  control  group. 


Table  4 

Standardized  Multiple  Regression  Weights 
Between  Predictor  and  Achievement  Variables 


Achievement  Variables 

Predictor  Variables 

TEXT 

JC 

IE 

HSRANK 

.13 

.07 

.14 

HOURS 

-.08 

-.07 

.00 

GPA 

.49 

.51 

.32 

CLASSIF 

.18 

.17 

.14 

TREAT 

.23 

.12 

.06 

Mult  R = 

.61 

.58 

.45 

10 


r-Tf 


Finally,  Table  5 shows  results  from  the  canonical  regression  when  all 
predictor  and  achievement  variables  were  considered  simultaneously.  Two 
statistically  significant  regression  equations  evolved.  The  standardised 
regression  weights  from  the  first  equation  indicate  that  GPA,  CLASSIF, 
and  TREAT  are  the  best  predictors  for  the  three  achievement  variables, 
which  all  had  weightings  in  the  same  direction.  This  equation  quantifies 
the  relationship  between  GPA  and  the  other  predictors;  it  shows  that  GPA 
is  about  2.5  times  more  powerful  than  any  other  variable  used  in  this  pre- 
diction system  for  predicting  overall  exam  performance. 

Table  5 

Standardized  Canonical  Regression  Weights 
of  Predictor  and  Achievement  Variables 


Equation 

Predictor 

Achievement 

HSRANK 

HOURS 

GPA 

CLASSIF  TREAT 

TEXT 

JC 

IE 

1 

-.20 

.11 

-.84 

-.30  .25 

-.56 

-.42 

-.18 

2 

.17 

.21 

-.02 

.09  -.86 

-.81 

.14 

1.01 

3 

.99 

.19 

-.67 

.22  .31 

.94 

-.13 

.51 

L , 

Equation 

1,  X2 

= 134.26,  df  = 

15,  p < .0001; 

.R  = 
c 

.65 

Equation 

2,  X2 

= 17.88,  df  = 

8,  p < .03  ; 

.R  = 
c 

.27 

Equation 

3,  X2 

= 1.84,  df  = 

3,  p < .25  ; 

.R  = 
c 

.09 

Equation  2,  which  represents  another  way  of  weighting  these  variables 
to  produce  a "reliable"  prediction  system,  shows  another  relationship  among 
the  variables.  First,  TREAT  is  by  far  the  most  powerful  predictor  and  second, 
TEXT  and  IK  are  the  two  achievement  variables  of  record.  This  relationship 
indicates  that  knowledge  of  whether  the  student  belonged  to  the  experimental 
or  control  group  is  a useful  predictor  of  performance  on  TEXT  and  IE,  in 
a correspondingly  inverse  direction.  The  sign  on  the  weightings  shows  that 
membership  in  the  experimental  group  (TREAT  = 2)  results  in  higher  TEXT 
scores  and  lower  IE  scores  than  membership  in  the  control  group  (TREAT  = 1). 
This  makes  sense,  when  considering  the  fact  that  the  Course  Management 
System  was  designed  to  facilitate  study  of  the  text  material,  and  that 
the  control  group  received  training  in  working  on  the  types  of  items  in 
the  IE  section  of  the  test. 

Student  Attitudes  Toward  the  Computer-based  Course 

As  indicated  previously,  only  319  of  the  original  350  experimental 
students  in  the  second  semester  completed  the  course.  These  students  were 
administered  a course  evaluation  questionnaire  via  PLATO.  The  responses 
to  the  five  items  listed  below  are  of  particular  interest.  As  shown, 
their  attitudes  were  quite  favorable. 
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1.  How  do  you  rate  the  course  in  general? 


Excellent  or  very  good  99 
Good  123 
Fair  86 
Poor  or  very  poor  11 


319 

2.  Would  you  take  another  course  taught  this  way? 


Strongly  agree  74 
Agree  129 
Disagree  67 
Strongly  disagree  49 


319 

3.  The  practice  quizzes  on  PLATO  were: 


Very  important/helpful  48 
Important  and  helpful  207 
Neither  important  or 

helpful  63 
Did  not  use  1 


319 

4.  The  opportunity  to  retake  examinations  was; 


Very  important/helpful  157 
Important  and  helpful  92 
Neither  important/helpful  21 
Did  not  use  49 


319 

5.  Everything  considered,  do  you  prefer: 


On-line  examinations 

191 

Pencll-and-paper  tests 

81 

No  preference 

47 

319 

The  marked  preference  of  students  to  take  on-line  examinations  indicated 
in  item  5 should  be  interpreted  in  light  of  the  circumstances  associated 
with  CAI  testing.  The  advantages  of  on-line  examinations  include  instant 
grading,  the  opportunity  to  retake  the  tests,  and  multiple  testing  times. 

The  disadvantages  include  machine  malfunctions,  time  pressure,  and  the 
inability  to  return  to  test  questions. 
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In  this  regard,  it  is  interesting  to  note  that  61  of  the  319  experimental 
students  elected  to  take  the  final  examination  prior  to  the  end  of  the  semester. 
The  fact  that  able  students  may  finish  early  may  be  an  advantageous  feature 
for  future  applications  of  CAI  in  which  time  savings  are  of  interest.  Finally, 
students  indicated  that  they  spent  on  an  average  of  4.6  hours  per  week 
studying  for  the  course. 

Student  Attitudes  Toward  Seminars 


The  students  generally  supported  the  seminars  that  were  designed  to  aid 
in  teaching  economic  concepts.  As  indicated  previously,  seminars  were 
evaluated  after  each  seminar  and  at  the  end  of  the  second  semester  using 
the  course  evaluation  questionnaire.  Students  were  asked  to  rate  the 
seminars  on  a five-point  scale.  Results  are  provided  in  Table  6. 


Table  6 

Open-Ended  Responses  Provided  by  Second  Semester  Experimental 
Students  on  Seminar  Approach 


Topic 

Number 

Means3 

S.D. 

Concept  of  Seminar 

42 

3.143 

1.424 

Grading  Procedure: 

Weight  in  Course 

7 

2.000 

1.000 

TA  Approach 

8 

2.000 

0.000 

Workload 

7 

1.714 

0.488 

Instructor 

12 

1.917 

1.084 

Con tent /Organization 

of  Seminar 

10 

2.600 

0.966 

Length  of  Seminar 

7 

2.286 

0.756 

Specific  Seminar 

23 

2.783 

1.622 

116 

aRated  on  a 5-point  scale. 

with  1 meaning 

highly  unfavorable  and  5, 

highly 

favorable. 


As  shown  in  Table  6,  the  concept  of  the  seminar  as  an  aid  to  under- 
standing received  the  highest  rating — 3.143.  However,  since  this  rating 
was  obtained  only  from  second  semester  students,  it  is  probably  lower  than 
it  might  have  been  during  the  first  semester,  because  the  instructors  re- 
duced the  grade  value  of  the  seminars  from  one-third  to  one-fifth  of  the 
final  grade.  In  spite  of  this,  66  percent  of  the  students  reported  that  the 
seminars  had  been  "very  helpful  or  very  important"  or  "important  and  help- 
ful." 
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The  seminar  evaluation  questionnaires  administered  to  students  after 
each  seminar  were  designed  to  evaluate  both  materials  and  Instructors. 
Instructor  ratings  were  made  on  eight  variables;  student  grades  provided 
a ninth.  Although  additional  seminars  were  taught  each  semester  by  student 
tutors  and  other  faculty  members,  they  were  not  evaluated. 

Mean  responses  provided  by  students  in  both  semesters  are  provided  in 
Table  7.  The  first  eight  variables  were  rated  on  Likert-like  scales,  with 
lower  ratings  usually  being  more  favorable  than  higher  ratings.  The  ninth 
variable  reflects  student  performance. 

Table  8 provides  results  of  an  analysis  of  variance  for  disproportionate 
cell  frequencies.  As  shown,  differences  between  semesters  were  significant 
for  variables  labeled  course  content,  course  in  general,  comparative  inter- 
est, and  progress  in  course.  For  each  significant  difference,  the  first 
semester  was  rated  more  favorably  than  the  second  semester,  except  for  in- 
structor number  5,  for  variables  labeled  course  content  and  progress  in 
course. 

A number  of  intangible  factors  may  have  been  responsible  for  these  sig- 
nificant differences  between  semesters.  For  example,  instructor  interest 
waned  somewhat  during  the  second  semester.  As  a result,  students  did  not 
find  cause  to  rate  the  instructor  lower,  the  course  more  difficult,  or  the 
workload  more  excessive,  but  they  were  less  satisfied  with  the  course 
content  in  general,  found  it  less  interesting,  and  made  less  progress. 

Note  that,  in  all  cases,  students  rated  the  seminar  less  interesting  in 
the  second  semester  (Table  8).  Perhaps  even  more  significant,  however, 
and  this  may  also  have  affected  the  instructors,  was  the  fact  that  the 
importance  of  the  seminar  had  been  played  down  considerably  during  the 
second  semester.  With  seminars  given  less  weight  and  thus  less  importance, 
students  may  well  have  found  it  less  appealing  and  interesting.  The  impor- 
tant factor  in  receiving  a good  course  grade  was  the  text  material;  seminar 
performance  was  not  vital.  This  point  was  further  supported  by  the  responses 
to  the  question  regarding  the  importance  of  and  request  for  open-ended 
comments  on  seminars  included  in  the  PLATO-administered  exam  given  to  ex- 
perimental students  in  the  second  semester.  Students  rated  the  concept 
of  the  seminar  quite  highly,  but  reacted  more  negatively  to  the  seminar 
weight,  workload,  and  length  (see  Table  6). 

Returning  to  Table  8,  we  find  that  differences  between  instructors 
were  significant  for  variables  labeled  major  instructor,  difficulty  level, 
comparative  interest,  and  seminar  grades.  Table  7 shows  that,  while  instruc- 
tors 1 and  3 fared  better  in  terms  of  overall  instructor  rating  and  compara- 
tive interest,  they  were  not  rated  as  the  instructors  offering  the  least 
difficult  seminars  or  higher  grades.  Since  these  findings  show  that  students 
did  not  rate  instructors  highly  because  of  easy  seminars  or  easy  grading, 
it  appears  that  there  were  significant  differences  between  instructors, 
in  spite  of  the  similarity  in  preparation  provided  by  training  sessions. 

These  differences  might  be  due  to  differences  in  teaching  experience — in- 
structors 1 and  3 entered  the  project  with  5 years  of  training  experience; 
and  the  other  three,  with  1/2  year  to  2 years  of  experience. 


Table  7 


Responses  Provided  by  CA1SMS  Students  on  Seminar 
Evaluation  Questionnaires 


Means 


Instructor  Code 


Fall  Semester 


Spring  Semester 


Course  Content  (Seminar)' 


1 

2 

3 

4 

5 


3.102 

2.790 

3.018 

3.068 

2.718 


2.583 

2.350 

2.977 

2.533 

3.075 


aRated  on  a 6-point  scale,  with  1 meaning  excellent  and  6,  very  poor. 

b Rated  on  a 4-point  scale,  with  1 meaning  very  difficult  and  4,  very  easy. 

cRated  on  a 5-point  scale,  with  1 meaning  excessive  and  5,  very  light. 


Table  7 (Continued) 


Instructor  Code 

Means 

Fall  Semester 

Spring  Semester 

Comparative  Interest^ 

1 

1.156 

1.623 

2 

1.640 

1.923 

3 

1.338 

1.453 

4 

1.712 

2.098 

5 

1.512 

1.845 

6 

Progress  in  Course 


1 

1.632 

2.223 

2 

1.615 

1.823 

3 

1.915 

2.057 

4 

1.852 

2.410 

5 

1.920 

1.895 

Class  Participation^ 


1 

1.120 

1.277 

2 

1.230 

1.417 

3 

1.133 

1.173 

4 

1.182 

1.170 

5 

1.064 

1.063 

o 

Seminar  Grades” 


1 

7.394 

6.277 

2 

6.753 

5.567 

3 

7.442 

7.297 

4 

7.577 

7.665 

5 

7.328 

7.230 

^Rated  on  a 3-point  scale,  with  1 meaning  more  interesting  and  3,  less 


interesting. 

0 

Rated  on  a 3-point  scale,  with  1 meaning  yes  (I  am  getting  behind  in 
my  textbook)  and  3,  no  (I  am  not  getting  behind). 

fRated  on  a 3-point  scale,  with  1 meaning  yes  (I  had  sufficient  opportunity 
to  discuss  and  ask  questions)  and  3,  no  (1  did  not  have  sufficient  oppor- 
tunity) . 

^Reflects  mean  score  obtained  by  CAISMS  students. 
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Table  8 


Disproportionate  Cell  Frequencies  Analysis  of  Variance 
on  Results  of  Seminar  Evaluation  Questionnaires 


Source 

df 

MS 

F 

P 

Course  Content  (Seminar) 

Instructor 

4 

0.4214 

1.66 

0.185 

Semester 

1 

1.1202 

4.40 

0.044* 

Interaction  (I  x S) 

4 

0.1740 

0.68 

0.609* 

Error 

31 

0.2545 

> 

Major  Instructor  (Seminar) 


Instruc  tor 

4 

1.2231 

. 5.35 

0.002* 

Semester 

1 

0.2574 

1.13 

0.297 

Interaction  (I  x S) 

4 

0.0887 

0.39 

0.815 

Error 

31 

0.2284 

• 

Course  in  General  (Seminar) 


Instructor 

4 

0.5059 

1.95 

0.127 

Semester 

1 

1.0844 

4.19 

0.049* 

Interaction  (I  x S) 

4 

0.1398 

0.54 

0.708 

Error 

31 

0.2591 

Difficulty  Level 


Instructor 

4 

0.1659 

2.87 

0.039* 

Semester 

1 

0.0054 

0.09 

0.763 

Interaction  (I  x S) 

4 

0.1672 

2.89 

0.038* 

Error 

31 

0.0578 

Workload 


Instructor 

4 

0.1960 

1.13 

0.361 

Semester 

1 

0.5410 

3.12 

0.087 

Interaction  (I  x S) 

4 

0.2929 

1.69 

0.178 

Error 

31 

0.1735 

Comparative  Interest 


Instructor 

4 

0.4160 

3.35 

0.022* 

Semester 

1 

0.9792 

7.88 

0.009* 

Interaction  (I  x S) 

4 

0.0337 

0.27 

0.894 

Error 

31 

0.1242 

*p  < .05 


l I 

il 
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Table  8 (Continued) 


Source  df  MS 


Progress  in  Course 


Instructor 

4 

0.1721 

1.06 

0.395 

Semester 

1 

0.8473 

5.20 

0.030* 

Interaction  (I  x S) 

4 

0.1415 

0.87 

0.494 

Error 

31 

0.1630 

Class 

Participation 

Instructor 

4 

0.0686 

1.70 

0.176 

Semester 

1 

0.0536 

1.33 

0.258 

Interaction  (I  x S) 

4 

0.0164 

0.41 

0.803 

Error 

31 

0.0404 

Seminar  Grades 


+ 2.5701  3.59  0.016* 

L 2.3552  3.29  0.079 

♦ 0.7229  1.01  0.417 

0.7156 


In  seminar  developmental  sessions,  an  attempt  was  made  to  equalize 
the  difficulty  level  and  workload  and  to  standardize  grading  procedures. 

The  staff  was  successful  in  equalizing  the  workload,  since  Table  8 shows 
no  significant  differences  on  this  variable.  However,  as  mentioned  above, 
there  were  significant  differences  in  difficulty  level  and  seminar  grades. 

The  former  might  stem,  to  a certain  extent,  from  the  concepts  dealt  with 
by  seminars.  The  latter  can  be  attributed  almost  exclusively  to  instructor 
2,  who  graded  considerably  lower  than  the  others.  Thus,  it  can  be  stated, 
with  some  reservation,  that  the  staff  was  successful  in  making  these  seminars 
fair  in  relation  to  the  equalization  of  difficulty  level  and  workloads 
and  standardization  of  grading  procedures. 

An  issue  that  is  very  interesting,  especially  in  terms  of  the  applica- 
bility of  these  seminars  to  other  courses,  was  chat  entitled  progress  in 
course.  This  variable  was  included  to  determine  whether  students  felt 
they  were  getting  behind  in  their  text  work  because  of  the  seminars,  which 
met  at  the  same  time  as  they  were  working  through  the  core  material. 

Table  7 distinctly  shows  that  students  in  the  first  semester,  whose  re- 
sponses tend  toward  "yes,"  felt  they  were  getting  behind  more  than  those 
in  the  second  semester,  whose  responses  tend  toward  "no."  This  finding 
is  significant  in  light  of  changes  in  seminar  scheduling  implemented  during 
the  second  semester.  As  indicated  previously,  to  ensure  that  students  main- 
tained progress  toward  completion  of  their  work,  deadlines  were  imposed. 

As  a result,  students  almost  always  entered  a seminar  just  after  completing 
a section  of  the  course  leading  to  a 1-hour  graded  exam.  This  meant  that  some 
seminars  met  on  an  irregular  basis  rather  than  regularly  over  a 4-week 
period.  Table  9 provides  the  results  obtained  when  the  two  types  of 
seminars  were  compared  along  the  nine  variables  in  question.  As  shown, 
the  only  significant  difference  between  the  two  types  was  in  the  variable 
labeled  progress  in  text.  Thus,  it  appears  that  students  felt  they  were 
further  behind  in  their  work  when  there  was  no  course-imposed  discipline 
than  when  there  were  deadlines.  It  is  not  clear  whether  or  not  the  seminar 
caused  the  students  to  fall  behind;  they  may  have  been  behind  all  along. 

In  any  event,  when  one  considers  the  student  responses,  it  seems  that  the 
concentrated  seminar  does  put  a burden  on  the  students.  Deadlines  help 
to  keep  them  from  falling  behind,  but  the  added  workload  of  a 4-week 
seminar  does  impose  a burden. 

In  concluding  this  section,  it  is  important  to  stress  a general  student 
approval  of  seminars.  They  proved  to  be  a vital  aspect  of  the  CMS,  by 
providing  students  with  a more  concentrated,  practical  orientation  than 
the  traditional  haphazard  "outside  readings."  Also,  the  idea  of  using 
seminars  as  a part  of  the  introductory  survey  courses  is  appealing  to 
students  who  often  find  difficulty  in  applying  theoretical  concepts.  Al- 
though the  exact  nature  of  the  seminar  presentation  is,  to  a large  extent, 
dictated  by  the  structure  of  the  course,  results  of  this  study  tend  to 
prove  that  seminars  are  worthwhile  and  deserve  further  utilization  and 
study. 
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Table  9 


Results  of  Comparison  Between  Types  of  Seminars 


Type 

Means 

t 

df 

P 

£ 

Course  Content 

Split  Seminars 

2.889 

1.325 

.8 

.20 

Continuous  Seminars 

2.543 

Seminars  in 

General3 

Split  Seminars 

3.003 

1.421 

18 

.17 

Continuous  Seminars 

2.638 

Difficulty  Level^ 

Split  Seminars 

2.523 

1.363 

18 

.24 

Continuous  Seminars 

2.362 

Q 

Work  Load 

Split  Seminars 

2.747 

0.124 

18 

.90 

Continuous  Seminars 

2.780 

Interest 

Level ^ 

Split  Seminars 

1.869 

1.435 

18 

.17 

Continuous  Seminars 

1.619 

e 

Progress  in  Text 

Split  Seminars 

2.323 

2.498 

18 

.03* 

Continuous  Seminars 

1.839 

Rated  on  a 6-point 

scale, 

with  1 meaning  excellent  and  6,  very  poor. 

^Rated  on  a 4-point 

scale, 

with  1 meaning  very  difficult  and  4,  very  easy. 

Q 

Rated  on  a 5-point 

scale. 

with  1 meaning  excessive  and  5,  very  light. 

^Rated  on  a 3-point 

scale. 

with  1 meaning  more  interesting  and  3,  less 

interesting. 

0 

Rated  on  a 3-point 

scale, 

with  1 meaning  yes  (I  am  getting  behind 

in 

my  textbook)  and  3, 

no  (I 

am  not  getting  behind). 

CONCLUSIONS 


'L'he  CMS,  with  minimal  help  from  the  instructors,  was  as  effective  as 
the  more  traditional  classroom  techniques.  It  is  essential  to  note  that 
the  computer  was  being  used  not  so  much  as  an  instrument  of  instruction, 
but  as  a director  and  monitor  of  student  progress.  As  has  been  demonstrated 
repeatedLy  in  courses  using  self-paced  instruction,  students  can  master 
subject  matter  "on  their  own,"  although  typically  with  a high  dropout  rate, 
an  unknown  amount  of  energy  expended  on  their  part,  and  a great  deal  of 
work  on  the  part  of  the  instructor.  As  used  in  this  experiment,  the  computer 
was  an  effective  device  for  managing  students,  for  keeping  records,  and 
for  providing  direct  communication  and  interaction  between  student  and 
instructor. 


The  strategy  adopted  in  this  experiment  of  systematically  quizzing 
students  over  standard  text  material  required  a relatively  small  commit- 
ment of  staff  time  to  programming  (see  Allessi  et  al.,  1974)  and  a rela- 
tively small  investment  of  student  time  in  interaction  with  the  computer 
(about  1 hour  each  week).  Student  reaction  to  the  numerous  small  quizzes, 
the  three  computer-administered  hour  examinations,  and  the  course  in  general 
was  favorable. 

What  is  perhaps  of  greater  significance  is  that  the  time  saved  by  the 
teaching  assistants  through  computer  management  could  be  used  to  teach 
short  seminars  on  topics  of  current  interest.  The  seminars  were  smoothly 
implemented,  appeared  to  make  economic  theory  more  understandable  via 
practical  situations,  and  maintained  high  student  interest. 
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RECOMMKNDATTONS 

The  research  and  development  reported  here  demonstrated  that  a standard 
course  could  be  individualized  relatively  simply,  managed  by  computer,  and 
maintain  or  somewhat  enhance  st<  lent  performance,  while  freeing  instructor 
time  for  seminars  and  tutoring.  If  the  system  could  be  adapted  successfully 
for  military  technical  training,  many  courses  could  be  converted  to  an  in- 
dividualized format,  thus  saving  student  time  and  allowing  more  effective 
use  of  instructors  as  tutors  and  as  supervisors  of  hands-on  training.  There- 
fore, it  is  recommended  that  a CMS-type  system  be  field  tested  in  a military 
setting  for  further  test  and  evaluation.  This  effort  should  be  designed 
to  test  the  applicability  of  the  methodology  in  converted  current  course 
materials  and  its  effectiveness  in  training  military  students,  while 
documenting  conversion  costs  and  cost  avoidance. 
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